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(57) Abstract 



A method and apparatus are provided for correcting for optical misalignment of the dual (left and right) images produced by a stereo 
electronic endoscope, or of the corresponding dual images produced by coupling electronic imaging devices to each of two monocular 
teaching ports of a stereo optical endoscope. The stereo image alignment technique comprises electronically capturing the video image data 
of the dual images (44R and 44L), and subsequendy processing that data electronically to correct for optical alignment errors (42). The 
rnethod involves digitizing (24R and 24L) the electronic data and digitally performing the equivalent of vertical image shift, and/or image 
size change, and/or image rotation as required to correct for any visual image misalignment. 
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Method And App aratus For Aligning Stereo Images 

Field of the invention 

This invention relates to a method and apparatus for aligning left and right 
images generated by an electronic stereo camera and more particularly to 
stereo endoscopes and operating microscopes. 

Background of the invention 

A three-dimensional ("3D") vision endoscope or microscope, hereinafter 
referred to as a "stereo" endoscope or microscope, comprises a stereoscopic 
optical system for forming a 3D image of an object. The stereoscopic optical 
system comprises objective lens means arranged to pick up two slightly 
different images of a object that approximate the two views (left and right) 
provided by human binocular vision. The difference between the two views is 
known as parallax. By properly aligning and presenting the two optical target 
images to human binocular vision, a three-dimensional sense of the target is 
presented to the viewer. 

In the case of a conventional (non-electronic) optical endoscope or 
microscope the two optical target images picked up by the objective lens 
means are passed to a viewing system in the form of a binocular eyepiece 
unit comprising two like optical channels each having, inter alia, a focusing 
lens followed by an ocular lens. In the case of an electronic stereo 
endoscope or microscope, the images picked up by the objective lens means 
are passed to an electronic stereo camera. As used herein the term 
"electronic stereo camera" is intended to denote a camera comprising two 
solid state electronic imaging devices, typically charge coupled devices 
(CCD's), that are capable of generating electrical output signals 
representative of images received thereby. The camera may also include 
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electronic circuit means for controlling operation of the imaging devices so as 
to produce said output signals. Alternatively, some or all of said electronic 
circuit means may be disposed outside of the camera. The camera may also 
comprise image formation optics for relaying images and focusing them on the 
imaging devices. Also as used herein the term "video camera head" is 
intended to designate a camera having a single electronic imaging device, 
with or without associated electronic circuit means for controlling its operation 
so as to produce a useful output signal representative of the images received 
by said imaging device. Hence a stereo video camera essentially comprises 
two video cameras heads. The video camera head also may also comprise 
image formation optics for relaying images and focusing them on the imaging 
device. 

The output signals of the electronic imaging devices are processed to 
provide video signals that in turn are used to drive an electronic viewing 
(display) system. The latter may comprise a conventional video monitor that 
is operated so as to provide a stereoscopic display in response to the video 
signals, the stereoscopic display typically being generated so as to be 
viewable by special polarized spectacles or, in the case where left and right 
images are displayed alternately, by spectacles having shutters that 
alternately block each eye in synchronism with alternate displaying of the left 
and right images. The viewing system also may be a head-mounted display 
unit, comprising first and second miniature electronic display devices, one for 
each eye, that display the left and right images respectively. The form of 
display system used is not critical to the invention. 

The capability of the human brain to fuse two images to produce a three- 
dimensional visual effect is tolerant of misalignment errors between the two 
images. However, only a small degree of misalignment, in the form of 
differences in image size, vertical position and rotation, will cause viewer 
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discomfort, and such discomfort increases as a function of the degree of 
misalignment. Consequently prolonged viewing of misaligned stereo images, 
such as occurs in cardiac surgery or pneurosurgery operations, can produce 
a high degree of viewer discomfort. 

In the optical domain, it is possible to align images by physically moving 
lenses, prisms, CCD's and other optical devices. However, in the case of a 
stereo endoscope the size limitation on the insertion portion of the endoscope 
results in the optical elements being quite small, making it difficult to 
mechanically adjust components to correct for misalignment errors. 

U.S. Patent 5,577,991, issued 26 November 1996 to N. Akui et al, 
discloses various methods for adjusting left and right images to correct for 
mismatch. Fig. 3 of that patent discloses a method and apparatus for 
electronically correcting for optical mismatch errors, while other figures of that 
patent show mechanical means for optical systems of images. Mechanical 
methods and means for correcting for misalignment errors of the type 
described are expensive and are not fully satisfactory for a number of 
reasons. The electronic method embodied in the system of Fig. 3 of said 
patent is not explained or illustrated in specific detail but appears to have 
certain limitations. Nevertheless, it is recognized that the equivalent of 
horizontal and vertical image shift, image size change, and image rotation can 
be performed very effectively in the electronic domain. 

Objects A nd Summary Of The Invention 

Accordingly, the primary object of this invention is to provide a new and 
improved method and means for electronically correcting for misalignment of 
stereo images generated by solid state imaging devices in electronic video 
cameras, especially where such cameras form part of or are coupled to stereo 
endoscopes and operating microscopes. 
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Another object is to provide a method and apparatus for correcting for 
stereo images misalignment that differs from and improves upon the 
electronic system shown in Fig. 3 of said U.S. Patent No. 5,577,991. 

A further object is to provide a new and improved stereo image alignment 
technique for endoscopes and operating microscopes which involves 
capturing electronically generated left and right images which may contain 
residual optical alignment errors and subsequently processing those images 
electronically to correct those errors. 

Still another object is to provide a method and apparatus for correcting 
stereo image misalignment problems which involves automatically capturing 
left and right electronic images (video image data), digitizing those images, 
processing the digitized images to achieve alignment, and processing the 
aligned digitized images to generate signals for driving a stereo display 
device. 

These and other objects are achieved by passing left and right channel 
video signals derived from first and second CCD's to analog-to-digital (A/D) 
signal converters to generate digitized video image data, inputting the 
digitized images to a stereo image processor that comprises first and second 
frame memory units for storing the video image data, and an image analysis 
computer for comparing the digitized images inputted to said first and second 
frame memory units, using said computer to compare the digitized left and 
right images inputted to said first and second frame memory units and to 
determine the sense and degree of mismatch between said digitized left and 
right images, generating correction command signals according to the sense 
and degree of mismatch between said digitized left and right images, inputting 
those command signals to said frame memory units as readout control signals 
to control the readout of video image data from said frame memory units, 
passing the digitized data read out from said memory units to digital to analog 
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converters so as to generate analog video signals, and passing said analog 
video signals to a stereo display device. By way of example, the preferred 
embodiment of the invention may be used in conjunction with a head-mounted 
binocular electronic display, but ihe stereo display device may also be a TV- 
type monitor. Also preferably in the case of an endoscope, the video signals 
are derived from a video camera that comprises an integral part of an 
electronic stereo endoscope. An alternative embodiment involves a stereo- 
optical operating microscope having two auxiliary observation ports, one for 
each optical channel, which are commonly identified as "teaching ports". A 
monocular electronic video camera head is coupled to each observation port, 
with the left and right video image signals derived from the two monocular 
camera heads being used drive a display after being processed to correct 
stereo misalignment errors. 

The Drawings 

Fig. 1 is a diagrammatic illustration of the general arrangement of an 
endoscope system and an image alignment measuring and adjusting system 
embodying the present invention; 

Fig. 2 is a diagrammatic illustration similar to Fig. 1 but providing details of 
the image alignment measuring and adjusting system. 

Fig. 3 illustrates certain details of the image capture memory shown in Fig. 

2; 

Figs. 4-7 illustrate the concept of digital electronic image processing and 
particularly the concept of separating data entry and data readout addressing 
sequences so as to effect corrections of various forms of misalignment; 

Fig. 8 illustrates a target used in measuring and correcting for image 
misalignment; 
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Figs. 9A and 9B constitute a computer software flow diagram indicating 
one form of computer program required to correct misalignment problems 
using the system shown in Figs. 1-3; 

Fig. 10 illustrates a stereo operating microscope having video camera 
heads coupled to each of its two teaching ports, the image alignment 
measuring and adjusting system of Fig. 2 making it possible to align the 
images formed by the two video camera heads to provide an electronically 
generated stereo display based on the right and left images transmitted by the 
microscope's two optical systems to the two cameras. 

In the several figures, like components or elements are designated by like 
numerals. 

Detailed Description Of Invention 

Fig. 1 illustrates diagrammatically a preferred form of image alignment 
measuring and adjusting system embodying the invention in combination with 
an electronic stereo endoscope. The latter comprises a housing 2 having a 
tubular insertion portion or barrel 4 containing two objective lenses 8R and 8L 
and two sets of relay lenses 10R, 10L. The housing also contains two pairs of 
45° mirrors 12R, 12L and14R, 14L, and a pair of focusing lenses 16R, 16L 
The foregoing optical components form two observation optical systems or 
channels, one right ("R") and the other left fL M ). The endoscope also 
contains a stereo video camera comprising two electronic imaging devices 
18R and 18L, e.g., two CCD's, positioned to pick up left and right images with 
parallax between them that are focused by lenses 16R, 16L Alternatively and 
preferably, each imaging device may be an RBG imaging device (not shown) 
that comprises a color resolution prism, an R-imaging CCD, a G-imaging 
CCD, and a B imaging CCD (see Figs. 11 and 31 of U.S. Patent No. 



SUBSTITUTE SHEET (RULE 26) 



WO 99/37098 



PCT/US98/27872 



5,579,772 for illustrations of RBG imaging devices coupled to a color- 
resolving prism. 

Although the camera is shown as built into housing 2 as an integral pari of 
the endoscope, it is to be understood that the camera may be constructed as 
an independent stereo unit, or as two independent monocular video cameras, 
that are permanently or releasably attached to the endoscope, in the manner 
illustrated in Figs. 7 and 8 of U.S. Patent 5,577,991. The significant thing is 
that the endoscope comprises two optical channels or image formation 
systems and two imaging devices 18R and 18L. The output terminals of the 
imaging devices are connected to control and signal processing circuits 22R 
and 22L. The latter drive the CCD units and process the electrical signal 
outputs from those units so as to generate video signals. In passing through 
units 22R and 22L the output signals from the CCD's are amplified and 
linearized and then analog video output signals representing sequential 
frames of the images seen by the CCD's are produced from those amplified 
and linearized signals. Such control and signal processing units are well 
known to persons skilled in the video art. In the case of RGB imaging 
devices, the control and signal processing circuits 22R and 22L amplify and 
process the individual R, G and B output signals and generate therefrom an 
RGB video signal. The video signal outputs from the control and signal 
processing circuits 22R and 22L are then passed to analog-to-digital (A/D) 
converters 24R and 24L where they are digitized. The digital output from 
converters 24R and 24L are then applied to a stereo image processor module 
26 designed according to this invention which looks for and measures 
misalignment errors between the R and L images represented by the digitized 
video signals, and then corrects for any discerned misalignment errors. The 
digital signal outputs of image processor module 26 representing the 
corrected digitized video images are then passed to digital-to-analog 
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converters 28R and 28L, and the analog video signal outputs from those 
converters are fed to a video processor unit 30 which comprises circuits for 
processing and formatting the R and L video signals for use in driving a 
stereo video display unit 32. The circuits required for video processor 30 are 
well known to persons skilled in the video art, as exemplified by U.S. Patent 
No. 4,253,447 issued to W.C. Moore et al for "Color Endoscope With Charge 
Coupled Device And Television Viewing". 

Fig. 2 presents further details of the preferred form of the image processor 
module 26. Although the AID converters 24R and 24L and the D/A converters 
28R and 28L are shown as discrete components separate from module 26, it 
is to be appreciated that they may be embodied in and form part of the image 
processor module itself. The latter comprises two dual-port image frame 
memory units 34R and 34L, with memory writing control circuits (i.e., writing 
address sequence generators) 36R and 36L coupled to one port and memory 
readout control circuits (i.e., readout address sequence generators) 38R and 
38L coupled to the second port. Dual port memory units offer the advantage 
that data can be read out from one location while simultaneously new data is 
entered into any other location. The memory writing control circuits 36R and 
36L control the entry of data from the A/D converters 24R and 24L and cause 
memory units 34R and 34L respectively to receive (read) such data on a 
frame-by-frame basis. The memory readout control circuits 38R and 38L 
cause data representing each image frame to be read out of the two memory 
units 34R and 34L respectively on a frame by frame basis in accordance with 
image correction instructions received from an image analysis computer 42. 
The digitized image data read out from memory units 34R and 34L are fed to 
the D/A converters 28R and 28L respectively. Additionally the same image 
data is fed to image capture memory units 44R and 44L, which in turn read 
that data out to computer 42 as hereinafter described. It is to be noted that 
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control and signal processing circuits 22R and 22L produce control signals 
which are applied to memory writing control circuits 36R and 36L, memory 
readout control circuits 38R and 38L, and image capture memory units 44R 
and 44L, whereby those circuits are clocked in synchronism with each image 
frame represented by the video signal outputs from signal processing circuits 
22R and 22L respectively. This control and clocking arrangement is 
represented in Fig. 2 by lines connecting the input line to A/D converters 24R 
and 24L to memory control circuits 36R and 36L, memory readout control 
circuits 38R and 38L, and image capture memories 44A and 44L. 

Fig. 3 illustrates details of image capture memory units 44R and 44L. 
These units comprise random access memories 48R and 48L, image capture 
control circuits 50R and 50L (i.e.. writing address sequence generators) and 
analysis readout control circuits 52R and 52L (i.e., readout address sequence 
generators). The input lines for memories 48R and 48L labeled "video output" 
are connected to the output lines of dual-port frame memory units 34R and 
34 L respectively. The clocking of image capture memories mentioned above 
is achieved by applying the control signals produced by signal processing 
circuits 22R and 22L to image capture control circuits 50R and SOL and 
analysis readout control circuits 52R and 52L. Readout of image data from 
memories 48R and 48L into computer 42 is controlled by readout commands 
provided by image analysis computer 42 via lines 54R and 54L (Figs. 2 and 
3). 

The invention is based on the concept that stereo images misalignment in 
electronic endoscopes and microscopes involves one or more of the 
following: (1) vertical shift, i.e., one image is slightly above or below the 
other, (2) one image is rotated relative to the other, and (3) one image is too 
large or too small relative to the other. These errors can be corrected by 
separately storing the two digitized images in memory, then reading out all or 
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only a portion of the data representing each image in a selected order 
calculated to properly align the two images, and then by storing the 
appropriate alignment compensation values into the stereo image processor's 
memory readout controls 38R and 38L. 

Figs. 4-7 illustrate the concept of digital electronic image processing, and 
particularly the concept of separating data entry and date readout addressing 
sequences. These figures illustrate a section of an image stored in memory 
unit 34L or 34R that is 8 lines by 16 pixels. The squares represent memory 
locations while the line and pixel numbers represent both data entry 
addressing sequences and date readout (in bold numerals) addressing 
sequences. The illustrations could equally well apply to video data storage 
"by the frame" or "by the field". 

Fig. 4 illustrates data being read in and read out with identical addressing 
sequences. Hence data entered from input line 1 is read out as output line 1, 
with identical alignment of pixel positions on each line. Consequently a 
readout from memory of the image data stored in memory unit 34R or 34L as 
illustrated in Fig. 4 produces an output exactly identical to the input data, and, 
therefore, the video data output from the memory unit exactly equals the video 
input to that memory. 

Fig. 5 illustrates data entry and data readout with slightly different 
addressing sequences. In this case assume that digitized image data was 
read into the memory unit exactly as shown in Fig. 4. However, data is read 
out in a different sequence. More specifically, data entered from input line 1 
is not read out at all, and the same is true of the first 3 pixels from all 
subsequent lines. Thus the read out of data from memory produces an output 
that is not equal to the data input. Essentially the output video data is shifted 
vertically up 1 line and horizontally left 3 pixels. This capability to move the 
output image signal generated by one imaging device in a way different from 
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the output image signal of another imaging device is the fundamental basis of 
the image alignment technique of this invention. 

Fig. 6 illustrates another mode of addressing the memory units. In this 
case data is entered into and read out of a memory unit with radically different 
addressing sequences in order to achieve a magnification of a portion of a 
video image. Again assume that digitized image data was read into the 
memory unit exactly as shown in Fig. 4. However readout begins with pixel 4 
of line 2. The second data readout is again pixel 4 of line 2. The same 
double readout occurs vertically, as output lines 2 and 3 both are read from 
data captured from input line 3. Thus the output video produced is generated 
from only a portion of the input image captured. In fact the output video 
generated with these addressing sequences is a linear 2:1 (4:1 area) 
magnification. 

The simple repetition of successive pixels and successive lines would 
create a magnified blotchy representation of the original input image. 
However, this blotchiness can be reduced substantially by a process called 
interpolation. In such process, output data values are generated by 
considering not only which pixels and which lines are being read out for any 
given output pixel, but also the values of other pixel data nearby. In particular 
each output address can be viewed as an integer part and a fractional part in 
both vertical and horizontal directions. The integer parts can be used to 
select which input pixel data points shall participate in a mathematical 
function to generate the output data, while the fractional parts can be used to 
weight the contribution of the various selected input pixels. 

Fig. 7 illustrates rotation as well as magnification of image data. In this 
case, data is read out so that it performs a linear 2:1 (4:1 area) magnification 
of the input data, but also a rotation of the output video image relative to the 
input video image. It should be noted also that as an output line is traversed, 
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the input line required for providing its data shifts in midline. This arbitrary 
sequence shift from input to output is the reason that the readout addressing 
mechanism must permit random access. 

In the practice of this invention, it is preferred that misalignment errors be 
corrected by changing only the R or the L video image according to the 
foregoing technique, while processing the other image unchanged. However, 
the system may be programmed so that in the event of relatively large 
misalignment errors, the correction be effected by changing each of the two 
video images by one-half of the total amount of change required to achieve 
correct alignment. 

Fig. 8 and the following description explain how stereo-pair misalignment 
can be measured and corrected by this invention. Fig. 8 illustrates a 
calibration target used to measure misalignment, the target comprising a grid 
of horizontal and vertical lines. The image analysis computer 42 is 
programmed so as to carry out the following procedure for measuring and 
correcting for stereo-pair misalignment using the target of Fig. 8. The 
procedure is as follows: 

1. The endoscope is mounted so that it is pointed at the target shown in Fig. 
8, with the target located at the natural convergence distance for the R and L 
optical observation channels of the endoscope, as represented in Fig. 1 by 
target T. This is the distance at which the two images should be the most 
alike. In fact if there are no alignment errors, they will be identical. 

2. The computer 42 sets the input and output channel address sequence 
generators 36R, 36L, 38R, and 38L for both the left and right channels to 
identical nominal conditions, with the result that the digitized video output of 
both memory units will be as illustrated in Figure 4. 

3. The computer 42 causes memory units 34R and 34L to capture R and L 
images of the entire target, or preferably an appropriate subset of both 
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images consisting of the four lines HI, H2, VI & V2 using a subsystem such as 
illustrated in Figure 8. In this subsystem, the data corresponding to light 
intensity along lines HI, H2, VI & V2 are available, and it is possible to 
associate the position of analysis output data points with image pixel position 
and line position. 

4. The computer determines the exact values of the intersections listed in 
Figure 8, namely: Lh11, Lh12, Lh13, Lh21, Lh22, Lh23, Lv11, Lv12, Lv13, 
Lv21, Lv22, and Lv23 for the Left image, and Rh11, Rh12, Rh13, Rh21, Rh22, 
Rh23, Rv11, Rv12, Rv13, Rv21, Rv22, and Rv23 for the right image. Since 
the target has been positioned such that the two images are supposed to be 
identical, the differences between left and right intersection values are used 
to determine misalignment between left and right images. If desired for 
maximum accuracy, the computer may be programmed to apply various 
techniques for noise cancellation, data filtering and line center calculations. 
With such techniques sub-pixel accuracies are possible. 

5. The computer calculates image size ratio as follows: 

SIZE_RATIO = (Lh1 3-Lh1 1 ) + <LH23-Lh21 ^ 
(Rh13-Rh11) + (Rh23-Rh21) 
The result of this calculation is applied to adjust one or both of the dual-port 
memory data readout sequence generators 42R and 42L, so that subsequent 
use of the two dual-port memories will result in equal size images of the same 
target. 

6. The computer calculates image rotation of each target image as follows: 

LEFT_ANGLE = arctan ( (Lh21+ Lh22 + Lh23 - Lh1 1 - Lh12 - Lh13) 

3(Y2 - Yl) 

RIGHT_ANGLE = arctan ( ( Rh21 + Rh22 + RH23 - Rh1 1 - Rh1 2 - Rh1 3) 

3(Y2 - Yl) 



SUBSTITUTE SHEET (RULE 26) 



WO 99/37098 



PCT/US98/27872 



14 



where Y2 and Yl are the line numbers for lines H2 and HI respectively The 
difference between Left.Ang.e and Righ L Ang,e is the rotation misalignment 
between left and right images. These factors are applied to adjust the dual- 
port memory readout sequence generators so that the resultant images will 
have equal rotation angles. 

7. Once size and rotation adjustments have been accomplished a second 
-mage capture is performed automatically. As a result of the size and rotation 
adjustments, this second image has zero size and ang.e misalignment This 
second image is used to correct horizontal and vertical offsets. The computer 
calculates horizontal offset misalignment as follows: 

LEFT_HORIZ = (Lh11 + Lh12 + Lh13 + Lh21 f Lh22 f Lh23)/6 

RIGHT_HORIZ = (Rh11 + Rh12 + Rh13 + Rh21 + Rh22 + Rh23)/6 
The amount of horizontal misalignment is (LEFTHORIZ - RIGHT.HORIZ) 
This factor is applied to the dual port memory readout sequence generators to 
correct the horizontal error. 

8. The computer then calculates vertical offset misalignment as follows- 
LEFTVERT = (Lv1 1 + |_v12 + Lv13 + Lv21 + Lv22 + Lv23) / 6 
RIGHT.VERT = (Rv1 1 + Rv12 + Rv13 - Rv21 + Rv22 + Rv23) / 6 
The amount of vertical misalignment is (LEFTVERT - RIGHT.VERT) The 
compute apphes this factor to the dual-port memory readout sequence 
generators 42R and 42L to correct the vertical error. 

Figs. 9A and 9B together constitute a flow diagram illustrating the 
program required to execute the foregoing alignment procedure. The flow 
d.agram is se.f-exp.anatory to one skilled in the art of computer programming 

It .s to be noted that the invention may be practiced other than as 
described above. For one thing, the two dual port memory units 34R and 34L 
are not required, although their use is preferred. Thus it is contemplated to 
replace dual port memory unit 34R with two conventional random access 
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memory units connected in parallel, with one of the replacement memory units 
being used for writing in data at the same time that data is being read out of 
the other replacement memory unit. Of course, the other dual port memory 
unit 34L would be replaced by two like conventional random access memory 
units that are connected in parallel and also would write data in and read data 
out on an alternating basis relative to one another. 

It is to be noted also that the invention may be used with an endoscope or 
microscope that utilizes two objectives, one for each channel, instead of the 
single objective shown in Fig. 1 . 

The invention also has application to stereo operating (surgical) 
microscopes of the type having auxiliary viewing ports commonly called 
"teaching ports". Fig. 10 illustrates diagrammatically an operating microscope 
having a barrel 60 containing an objective lens 62 and right and left imaging 
channels comprising focusing lenses 64R and 64L. two beam splitters 66R 
and 66L, and two eyepieces 68R and 68L comprising lens pairs 70R and 70L 
forming a binocular viewing system. Barrel 60 has two side ports fitted with 
side tubes 72R and 72L. Two 45° mirrors 74R and 74L are mounted in the 
two tubes as shown. Attached to tubes 72R and 72 L and communicating with 
the two mirrors via apertures in the two tubes are video camera heads 78R 
and 78L comprising focusing lenses 80R and 80L and electronic imaging 
devices 82R and 82L each in the form of a single CCD or an RBG imaging 
unit as described above. Beam splitter 66R splits the image focused thereon 
by focusing lenses 64R, transmitting it to lens pair 70R and also to mirror 
74R, while beam splitter 66L splits the image focused thereon by lenses 64L. 
transmitting the image to ocular lens pair 70L and also to mirror 74L. As a 
result the binocular viewing system comprising lens pairs 70R and 70L 
provides one surgeon with a stereo image presentation, while camera head 
78R sees only the image carried by the right channel and camera head 78L 
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sees only the image carried by the left channel. However, according to this 
invention, the output image signals derived from camera heads 78R and 78L 
are fed to an image alignment system as shown in Figs, 1 and 2, with the 
signal outputs of imaging devices 82R and 82L being inputted to control and 
signal processing circuits 22R and 22L respectively. Since it is obviously 
difficult to assemble an operating microscope as shown in Fig. 1 0 so as to 
achieve correctly aligned stereo image pairs, applying the output of camera 
heads 78R and 78L to a system as shown in Figs. 1 and 2 will correct for any 
misalignment and provide a second surgeon with a stereo presentation via 
electronic display 32. 

It should be noted that the particular optical channels illustrated in Figs. 1 
and 10 are presented by way of example only, and may be replaced by other 
optical systems of like purpose. 

Thus it is contemplated that the microscope of Fig. 10 could be modified 
by replacing objective lens 62 with two smaller objectives, one for each 
channel as is common practice and as is the case with the endoscope 
represented in Fig. 1. The invention also is not limited to systems using a 
particular type of display device. Thus the display device may be a head- 
mounted unit or a conventional video monitor. The invention also can be 
used with display devices that are adapted to display the left and right images 
alternately or concurrently, according to well known techniques for presenting 
3-D pictures. 

It should be noted that the invention is not limited to stereo electronic 
endoscopes and microscopes but can be used for aligning the stereo images 
of any stereo electronic camera. Although the stereo image alignment 
components shown in Fig. 2 can be united as a separate test instrument for 
use by instrument manufacturers in aligning stereo microscopes, endoscopes, 
and stereo video cameras per se, the invention also may be incorporated as a 
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permanent part of the electronics system of a microscope or endoscope. In 
the case where the stereo camera is a permanent part of an instrument such 
as an endoscope, the stereo alignment system shown in Figs. 1-3 need not 
be incorporated into the instrument's electronic system, but instead the 
corrections determined to be necessary by computer 42 may be permanently 
stored in a non-volatile memory that is integrated into the instrument or its 
associated electronics, with the electronics programmed so that each time the 
instrument is activated the data recorded in the non-volatile memory will be 
accessed automatically and used to adjust the stereo images as fed to the 
display system. Furthermore, as suggested by the two broken lines labeled 
"digital output" in Fig. 2, the image alignment system of this invention may be 
modified by using the corrected digital video image data output from memory 
units 34R and 34L to drive a digital video display according to recently 
developed digital video techniques. 

The invention has numerous advantages. The electronic system for 
correcting misalignment errors utilizes conventional components and the 
software required to control operation of computer 42 and the other parts of 
stereo image processor 26 is straightforward and imposes no unreasonable or 
costly hardware requirements. Best of all, the alignment can be accomplished 
rapidly and accurately. 

Other modifications and advantages will be obvious to persons skilled in 
the art. 
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WHAT IS CLAIMED IS: 

1 . A stereo endoscope or microscope comprising: 

first and second optical channels for viewing an object*and forming and 
transmitting first and second images of said object having a parallax between 
them; 

first and second imaging means coupled to said first and second 
optical channels respectively for picking up said first and second images 
respectively and producing first and second output signals representative of 
said first and second images respectively; 

signal processing means for converting said first and second output 
signals to first and second video signals respectively; 

image-correcting means responsive to said first and second video 
signals for modifying said first and second video signals to correct for 
misalignment of said first and second images relative to one another; and 

an electronic display means responsive to said modified first and 
second video signals for reproducing said first and second images and 
displaying said reproduced images either alternately or concurrently for 
viewing as a 3-D image. 

2. The stereo endoscope or microscope according to claim 1 wherein said 
first and second video signals are analog video signals representative of said 
first and second images on a frame-by-frame basis, and further wherein said 
image-correcting means comprises: 

means for converting said first and second analog video signals to first 
and second digital data on a frame by frame basis; first and second memory 
units for receiving and storing said first and second digital data; write control 
means for controlling the writing of said first and second digital data into said 
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first and second memory units; readout control means for controlling the 
reading out of said first and second digital data from said first and second 
memory units; means coupled to said first and second memory units for 
driving said electronic display means in response to said first and second 
digital data read out from said memory units; image analysis means coupled 
to said first and second memory units for comparing said first and second 
digital data as read out from said memory units to measure any relative 
differences in the (a) sizes, (b) horizontal and vertical positions, and (c) 
rotational positions of said first and second images and for producing first and 
second correctional signals according to said relative differences; and means 
for applying said first and second correctional signals to said readout control 
means as readout commands so as to adjust the readout of digital data from 
at least one of said first and second memory units in a manner that eliminates 
said differences in the images represented by subsequent first and second 
digital data fed to said first and second memory units. 

3. The stereo endoscope or microscope according to claim 2 wherein said 
each of said first and second memory units is a dual port memory unit. 

4. A stereo endoscope or microscope apparatus comprising: 

a pair of optical systems for transmitting an image of an object as first 
and second images having a parallax between them; 

a pair of imaging means for picking up said first and second images as 
transmitted by said optical systems and producing first and second output 
s.gnals representative of said first and second images respectively; 

a display means for displaying said two images either alternately or 
concurrently for viewing as a 3-D image; 
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image-correcting means for correcting for misalignment of said first and 
second images relative to one another so that said images coincide with one 
another as displayed by said display means, said image-correcting means 
comprising first and second signal processing channels for processing said 
first and second output signals and producing first and second analog video 
signals therefrom, means for converting said first and second analog video 
signals to first and second digital data on a frame by frame basis; first and 
second memory units coupled to said first and second signal processing 
channels for receiving and storing said first and second digital data; write 
control means for controlling the writing of said first and second digital data 
into said first and second memory units; readout control means for controlling 
the reading said first and second digital data out from said memory units; 
means coupled to said first and second memory units for driving a stereo 
display device in response to said first and second digital data read out from 
said memory units; image analysis means coupled to said first and second 
memory units for comparing said first and second digital data as read out from 
said memory units to measure any relative differences in the sizes, horizontal 
and vertical positions, and rotational positions of said first and second images 
and for producing first and second correctional signals according to said 
relative differences; and means for applying said first and second correctional 
signals to said readout control means as readout commands so as to adjust 
the readout of digital data from at least one of said first and second memory 
units in a manner that eliminates said differences in images represented by 
subsequent first and second digital data fed to said first and second memory 
units. 

5. A stereo microscope comprising: 
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first and second optical channels for viewing an object and forming and 
transmitting first and second images of said object having a parallax between 
them; 

viewing means coupled to said first and second optical channels for 
displaying said first and second images for viewing as a 3-D image; 

first and second imaging means coupled to said first and second 
optical channels respectively for picking up said first and second images 
respectively and producing first and second output signals representative of 
said first and second images respectively; 

signal processing means for converting said first and second output 
signals to first and second video signals respectively; 

image-correcting means responsive to said first and second video 
signals for modifying said first and second video signals to correct for 
misalignment of said first and second images relative to one another; and 

an electronic display means responsive to said modified first and 
second video signals for reproducing said first and second images and 
displaying said reproduced images either alternately or concurrently for 
viewing as a 3-D image. 

6. A microscope according to claim 5 wherein said image-correcting 
means comprising first and second signal processing channels for processing 
said first and second output signals and producing first and second analog 
video signals therefrom, means for converting said first and second analog 
video signals to first and second digital data on a frame by frame basis; first 
and second memory units coupled to said first and second signal processing 
channels for receiving and storing said first and second digital data; write 
control means for controlling the writing of said first and second digital data 
into said first and second memory units; readout control means for controlling 
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the reading said first and second digital data out from said memory units; 
means coupled to said first and second memory units for driving a stereo 
display device in response to said first and second digiiai data read out from 
said memory units; image analysis means coupled to said first and second 
memory units for comparing said first and second digital data as read out from 
said memory units to measure any relative differences in the sizes, horizontal 
and vertical positions, and rotational positions of said first and second images 
and for producing first and second correctional signals according to said 
relative differences; and means for applying said first and second correctional 
signals to said readout control means as readout commands so as to adjust 
the readout of digital data from at least one of said first and second memory 
units in a manner that eliminates said differences in images represented by 
subsequent first and second digital data fed to said first and second memory 
units, whereby said reproduced images coincide with one another and are 
presented for viewing by said display means as a 3-D image. 

7. The microscope according to claim 6 wherein said first and second 
memory units are dual port memory units. 

8. An apparatus for correcting for misalignment of the separate left and 
right images of a stereo electronic camera having a first and second optical 
means for transmitting an image of an object as first and second images 
having a parallax between them, and first and second imaging means for 
picking up said first and second images as transmitted by said optical systems 
and producing first and second output signals representative of said first and 
second images respectively on a frame by frame basis, each frame 
constituting a single image of said object, said apparatus comprising: 



SUBSTITUTE SHEET (RULE 26) 



WO 99/37098 



PCT/US98/27872 



23 

first and second signal processing channels comprising means for 
generating first and second video signals in response to said first and second 
output signals respectively, said first and second video signals also being 
representative of said first and second images on a frame by frame basis; 

an electronic display means; 

display operating means responsive to said first and second video 
signals for causing said display means to reproduce said first and second 
images on a frame by frame basis and display said reproduced first and 
second images either alternately or concurrently for viewing as a 3-D image; 

image-correcting means for correcting for misalignment of said first 
and second images relative to one another so that said reproduced first and 
second images appear coincident with one another as a 3-D image as 
displayed by said display means, said image-correcting means comprising 
first and second means for converting said first and second video signals to 
first and second digital data representative of said first and second images 
respectively on a frame by frame basis, first and second memory units for 
receiving and storing said first and second digital data; write control means for 
controlling the writing of said first and second digital data into said first and 
second memory units; readout control means for controlling the reading out of 
said first and second digital data from said memory units; image analysis 
means coupled to said first and second memory units for comparing said first 
and second digital data as read out from said memory units to measure any 
relative differences in the sizes, horizontal and vertical positions, and 
rotational positions of said first and second images and for producing first and 
second correctional signals according to said relative differences; and means 
for applying said first and second correctional signals to said readout control 
means as readout commands so as to adjust the readout of digital data from 
at least one of said first and second memory units in a manner that eliminates 
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any of said differences in images represented by subsequent first and second 
digital data fed to said first and second memory units respectively. 

9. A method of correcting for misalignment of the separate left and right 
images of a stereo electronic camera having a first and second optical means 
for transmitting an image of an object as first and second images having a 
parallax between them, and first and second imaging means for picking up 
said first and second images as transmitted by said optical systems and 
producing first and second output signals representative of said first and 
second images respectively, said method comprising the steps of: 

processing said first and second output signals so as to produce first 
and second analog videp signals therefrom on a frame by frame basis; 

converting said first and second analog video signals to first and 
second digital data on a frame by frame basis; 

writing said first and second digital data to first and second memory 
units on a frame by frame basis; 

reading said first and second digital data from said first and second 
memory units on a like frame by frame basis; 

analyzing said read first and second data on a frame by frame basis to 
measure any relative differences in the sizes, horizontal and vertical positions 
and rotational positions of said first and second images and producing first 
and second correctional signals according to said relative differences; 

applying said first and second correctional signals to said readout 
control means as readout commands so that the readout of digital data from 
at least one of said first and second memory units representing subsequent 
successive frames of said first and second images is adjusted in a manner 
that eliminates said differences in said successive frames of said first and 
second images. 



SUBSTITUTE SHEET (RULE 26) 



WO 99/37098 



PCTAJS98/27872 



25 

1 0. The method of claim 9 further including the step of generating a visual 
display of said first and second images on a frame by frame basis in response 
io said first and second digital data read out from said memory units. 

1 1 . The method of claim 10 wherein said first and second digital data read 
out from said memory units also is converted to third and fourth analog video 
signals respectively, and said visual display is generated in response to said 
third and fourth analog video signals. 

12. The method of claim 9 wherein said stereo camera is part of an 
endoscope. 

13. The method of claim 9 wherein said stereo camera is part of a stereo 
microscope. 

14. The method of claim 9 wherein said memory units are dual port 
memory units. 



SUBSTITUTE SHEET (RULE 26) 



WO 99/37098 



1 / 11 



PCT/US98/27872 




WO 99/37098 



2 / 11 



PCT/US98/27872 




WO 99/37098 



3 / 11 



PCT/US98/27872 




LLI 




m 




ZD 


i 


E CAPT 


ONTRO! 


e> o 











CD 




O 
UJ 
Q 

> 





2 




II 












C9 O 




IMA 




CD 


to 



ZD 

o 



WO 99/37098 



4 / 11 



PCT/US98/27872 





CNJ 


CO 




LO 


CD 




GO 


LU 


LU 


LU 


LU 


LU 


UU 


LU 


LU 






Z 




2 


2 







CO 



to 



i — <e 
i 



i — , 



l 



CD 



CO I 



I v=- 



CO 



CO 



CO 



CO 



— utI — toi 

"I ^ I 



-~4 



L 

S2| 



I 

I 



Sh - ! "I \ . 



I 

_J_. 

CO , 



52 I 

I 

tor 
"~ i 



co 



CO 



—Oy J 



21 



._! 

CO I CO 



I 



c\TJ cm' 



I 



I 



.1 

I 

^+ 
" I 

,4- 



<2r 
i 



'cnT [ cnT 



h-co 



_| 



I 



£ I oo- 

Q- co i 



cd | 
I 



CD 



CD 



CD 



CD | 



CD 



CD 



oo 



OO 



CO 



I 

I 40 



__]. 

CO I 



— 1 



i 



CO 



-6sr 
i 



co 



CO 



CO 



I 



LO 



I « 



lO~| to! 



I — , 



^ I 



I 



I 
I 



co i 

I 

5f 



I 



CO 



<o, 

_l 



co 



L. 



eg 





- -1 


■ -4 


- -4- 


- 4-- 


4-- 


1-- 




CO 1 


CO, 


CO, 


CO, 


co. 




CO, 


co 




1 


1 


1 


1 


1 


1 




CM | 


CM | 


CM | 


CM | 


CM | 


CM | 


CM | 


CNI 


— I 


1 

1 


I 

^ 1 
1 


1 


1 
1 


1 


1 

- 1 




1 






1 


1 


1 


1 






CM 


CO 




WO 


CO 




OO 



LU 



WO 99/37098 



5 / 11 



PCI7US98/27872 



i— CM 



CO 



LO 



CD 



LU 



CD 



r 



CO 



I- 
I 

L 

I 
I 

r 

i 

I 

U. 



cr> 



D- OO | 



CO 



CO 



CO 



CO 



CO 



CO 



CM 



-r 
I 

—I- 



1 — 

CM j CM 

^ -4 " 



J CM | CM 



I 

r- 
i ^ 

4-- 



i 



CO 



I-- 
I ^ 



_"L_J__ J L _L J I 

I 



I 



I 



_L__± 



I 



I 



I 



CO 



CO 



[-- 

co i cn 

1 

OO I OO 



T 1 



r 



CD 



CO 



CO 



I 



4-. 
I 



I 



i 

I CO 

J — 



I — 
t-- 

I CO 



I 



I 



CO 



I 



CO 



I 



CO 



CO 



CO 



I 1 



r- I 



1- 

co l 

1- 

I. 



,-- 

LO I CO 

[_- 

I 



I 

T 1 — 

I LO I LO 

-+ — -f- 
I ^ I 

_L ±_ 



I 



CO 



CO 



1 — 
I » 
-I — 

J 



I — 
I « 

\ — 

I 

I 



CO 



CO 



CO 



r 



I 1 



1" 

CM , 

1- 

1. 



CO j CO 

[ 



I 



CO 



I 



CO 



CO 



CO 



CM 



CM 



-t- 
I - 

-4- 



T __ T _ 

| CM | CM 

-t — 1-- 



r 



CM 



l — 



CO 



L 



CM 



r 



~ I" 



-I- 
I 

_L 

I 
I 

T 



4- 
I 

_L 

I 
I 

T 

I 



I • 

4-. 

I 

J_ 

I 
I 

r 



u 
I 

L 



r 



i— CM 



CO 



LO 



CO 



CO 



UJ 



WO 99/37098 



6 / 11 



PCT/US98/27872 



CO 



LO 



1 1 



CO 



CM 



r 

u. 

r- 

L. 



i 1- 



LU 

>< oo 
Cl- 



io 



LO 



CO 



CM 



-I- 



-1 
J 



4- 

i 
i 



T 

I 

J 

• KM CO W- 
I ,1X1 LLi LU 



I 



r 



4 
i 
i 

T 



4- 

I 

I 

r 



i 1- 



_-L i 1^ 

LO |CO po CD fc- 
LJJ |LU LU ^JJ LU |UJ 

— 5 -{fj-L3-(3 -3-p3~3 -pJ-^J-fS- 

I I I 

_CO— JcQ— CO -Lo— CQ-JcQ. 
LQ |lO LQ |lO LO jtQ 

-^4*— - j^* — — y^£ * 

co |ro co |co co |ro 

— r r r 

- C= - cz - c= 



1 

I 

^~W~CO~ 



co Jco co Icq 
lo |lo lo |lo 

co |co CO |co 

r — r 
^ c= ^ c= 



xd Icq co 

LO jug LO 

co (no CO 

— r — 
^ e: ^ 



r 



i 



r 




<c- CM 



CO 



LO 



CO 



OO 



WO 99/37098 



7 / 11 



PCT/US98/27872 




I 1 



o> 



>< oo 



r 



^ i 



CO 



to 



ff. I£2 co to ^ 1 



CO 



w i$2 $2 to „ I ; - t 



I |-tu- 



59 



U- 



1 
I 

-J 



I 1 

I I 
I I 



9 ^^rJ-r^J * °° 0O i 
*-> — 1_ i i ^ 




I 
I 

— I 



I 
I 

T' 
I 

._]_. 



CNJ 



CO 



CO 



CO 



WO 99/37098 



8 / 11 



PCT/US98/27872 



CM 

> ^ 
CVI 



CO 

3E 



CNI 



CM 



I 



S3 



CM 
CM 



CM 



CO 



CO 



I 

CM 



WO 99/37098 



9 / 11 



PCT/US98/27872 




SET LEFT & RIGHT DATA READOUT 
TO NOMINAL CONDITIONS 



© 



SET COMMON LEFT & RIGHT TARGET 
AT NATURAL CONVERGENCE DISTANCE 



I 



CAPTURE LEFT & RIGHT IMAGES 



LOOP FROM 



STEP 10 



DETERMINE IMAGE INTERSECTION VALUES USING 

A) NOISE CANCELLATION 

B) DATA FILTERING 

C) LINE-CENTER PROCESSING 



LEFT LINE INTERSECTIONS RIGHT LINE INTERSECTIONS 



Lh11 


Lh12 


Lh13 


Rh11 


Rh12 


Rh13 


Lh21 


Lh22 


Lh23 


Rh21 


Rh22 


Rh23 


Lv11 


Lv12 


Lv13 


Rv11 


Rv12 


Rv13 


Lv21 


Lv22 


Lv23 


Rv21 


Rv22 


Rv23 



FIG. 9 A 



WO 99/37098 PCT/US98/27872 

10 / 11 



1 



CALCULATE IMAGE CORRECTION FACTORS 



CD 



I 



_ (Lh13-Lh11) + (Ui23-Lh21) 
SIZE RATIO - ( Rh13 . Rh11 ) + (Rh23-Rh2l) 



© 



I 



LEFT TARGET ANGLE = ARCTAN ( < Lh21 iJM^WI -Lh12-U,13 ) , 
RIGHT TARGET ANGLE = ARCTAN ( (Rh2URh22.RI^3-Rh11 -Rh12-Rh13) ) 
ROTATION DIFFERENCE = LEFT TARGET ANGLE - RIGHT TARGET ANGLE 



LEFT HORIZONTAL = (Lh1 1 + Lh12 + LH13 + Lh21 + Lh22 + Lh23) / 6 
RIGHT HORIZONTAL = (Rh1 1 + Rh12 + RH13 + Rh21 + Rh22 + Rh23) / 6 
HORIZONTAL DIFFERENCE = LEFT HORIZONTAL - RIGHT HORIZONTAL 



® 



LEFT VERTICAL = (Lv11 +Lv12 +Lv13 +Lv21 +Lv22 + Lv23) /6 
RIGHT VERTICAL = (Rv1 1 + Rv12 + Rv13 + Rv21 + Rv22 + Rv23) / 6 
VERTICAL DIFFERENCE = LEFT VERTICAL - RIGHT VERTICAL 



I 



® | APPLY NEW IMAGE CORRECTION FACTORS 



I 



30 DECISION TO REPEAT OPERATIONS 3-10??? 



I 



LOOP TO 



STEP 3 



ALIGNMENT COMPLETE 



FIG. 9B 



INTERNATIONAL SEARCH^&EPORT 



International application No. 
PCT/US98/27872 



A. CLASSIFICATION OF SUBJECT MATTER 
IPC(6) :H04N 13/00. 15/00; A61B 1/04 
US CL : 348/42, 45 • 

According to International Patent Classification (IPC) or to both national classification and IPC 


a FIELDS SEARCHED 


Minimum documentation searched (classification system followed by classification symbols) 
U.S. : 348/42, 45. 65. 66. 69. 70 


Documentation searched other than minimum documentation to the extent that such documents are included 
None. 


in the fields searched 


Electronic data base consulted during the international search (name of data base and, where practicable, search terms used) 
APS 


C. DOCUMENTS CONSIDERED TO BE RELEVANT 


Category* 


Citation of document, with indication, where appropriate, of the relevant passages 


Relevant to claim No. 


X 


US 5,577,991 A (AKUI et al) 26 November 1996, Fig. 1 and fig. 3 
and coL 1, lines 17-47, col. 6, lines 33-67 and col. 7, lines 7-11, 27- 
29. 


1-14 


A 


US 5,646,680 A (YAJIMA) 08 July 1997, see entire document 


1 


A 


US 4,873,572 A (MIYAZAKI et al) 10 October 1989, see entire 
document 


1 


] Further documents are listed in the continuation of Box C. j | See patent family annex. 


• Spec iui caiegoriei of cited document*: 

"A" document defining the general state of the art which is not considered 
to be of particular relevance 

"E" earlier document published on or after the international filing date 

*L" document which may throw doubts on priority daunts) or which is 
cited to establish the publication date of another ciution or other 
special reason tas specified) 

"O* document referring to an oral disclosure, use. exhibition or other 
means 

•P" document published prior to the international filing date but later than 
the priority date claimed 


•T" later document published after the international filing date or priority 
date end not in conflict with the eppUcation but cited to understand 
the principle or theory underlying the invention 

•X' document of particular relevance; the claimed invention cannot be 
considered novel or cannot be considered to involve an inventive step 
when the document u taken alone 

•Y" document of particular relevance; the claimed invention cannot be 
considered to involve an inventive step when the document is 
combined with one or more other such documents, such combination 
being obvious to a person skilled in the art 

document member of the same patent family 


Date of the actual completion of the international search 
01 MARCH 1999 


Date of mailing of the international search report 

22 APR 1999 


Name and mailing address of the ISA/ US 
Commissioner of Patents and Trademarks 
Box PCT 

Washington, D.C. 20231 
Facsimile No. (703) 305-3230 


Authorized officer -n 

TOMMY P. CHIN j^V^' 
Telephone No. (703) 305-4715 



Form PCT/ISA/210 (second sheet)(July 1992) * 



